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ABSTRACT 

An experimental telerobotics (TR) simulation is described suitable 
for studying human operator (H.O.) performance. Simple manipulator 
pick-and-place and tracking tasks allowed quantitative comparison of a number 
of calligraphic display viewing conditions. 

A number of control modes could be compared in this TR simulation, 
including displacement, rate and accelerator control using position and force 
joysticks. A homeomorphic controller turned out to be no better than 
joysticks; the adaptive properties of the H.O. can apparently permit quite 
good control over a variety of controller configurations and control modes. 
Training by optimal control example seemed helpful in preliminary experiments. 


■''Research staff , . 

"•"Students in graduate bioengineering class ME 210, Biological Control 

Systems, Fall 1985. 
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An introduced communi cati on delay was found to produce decrease in 
performance. In considerable part, this difficulty could be compensated for 
by preview control information. That neurological control of normal human 
movement contains a sampled data period of 0.2 seconds may relate to this 
robustness of H.O. control to delay. 

The Ames-Berkeley enhanced perspective display was utilized in 
conjunction with an experimental helmet mounted display system (HMD) that 
provided stereoscopic enhanced views. Two degree-of-freedom rotations of the 
head were measured with a Helmholtz coil instrument and these angles used to 
compute a directional conical window into a 3-D simulation. The vector 
elements within the window were then transformed by projective geometry 
calculations to an intermediate stereoscopic display, received by two video 
cameras and imaged onto the HMD mini -display units (one-inch CRT video 
receivers) mounted on the helmet. 
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INTRODUCTION 


A tel eroboti c , TR, system is defined as a distant robot with vision 
and manipulator and/or mobility subsystems controlled by a human operator, 

HO. The HO is informed mainly by a visual display , but also by other sensors 
and other sensory displays, i.e. auditory, force or tactile. His control can 
be direct via joysticks, or supervisory via command and control primitives 
effected by pari tally autonomous robotic functions. Delays and bandwidth 
limitations in communication are key problems, complicating display and 
control (Stark, Kim, Tendick, et al , 1986). 

The research presented here was initially carried out by the students 
taking a graduate control course, ME 210 "Biological Control Systems: 
Telerobotics. " 

EXPERIMENTAL SET-UP FOR THREE-AXIS PICK-AND-PLACE TASKS 


A teleoperation simulator constructed with a display, joysticks, and 
a computer enabled three-axis pick-and-pl ace tasks to be performed and various 
display and control conditions evaluated (Figure 1). A vector display system 
(Hewlett-Packard 1345A) was used for fast vector drawing and updating with 
high resolution. In our experiments, displacement joysticks were mainly used, 
although in one experiment a force joystick was used to compare with a 
displacement joystick. An LSI-1 1/23 computer with the RT-11 operating system 
computer was connected to the joystick outputs through 12-bit A/D converters, 
and to the vector display system through a 16-bit paraller I/O port. 
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A typical presentation on the display screen for three-axis 
pick-and-pl ace tasks included a cylindrical manipulator, objects to pick up, 
and boxes in which to place them, all displayed in perspective (Figure 2). 
Since perspective projection alone is not sufficient to present 
three-dimensional information on the two-dimensional screen, a grid 
representing a horizontal base plane and reference lines indicating vertical 
separations from the base plane are also presented (Ellis, Kim, McGreevy , 

Tyler and Stark, 1985; Kim, Ellis, Tyler and Stark, 1985). The human operator 
controlled the manipulator on the display using two joysticks to pick up each 
object with the manipulator gripper and place it in the corresponding box. 

One hand, using two axes of one joystick, controls the gripper position for 
the two axes parallel to the horizontal base plane (grid). The other hand, 
using one axis of the other joystick, controls the gripper position for the 
third axis (vertical height) perpendicular to the base plane. Picking up an 
object is accomplished by touching an object with the manipulator gripper. 
Likewise, placing an object is accomplished by touching the correct box with 
the manipulator gripper. 


Puma Arm Si mu! ator 

In addition to the cylindrical manipulator simulation, the kinematics 
and dynamics a six degree-of -freedom Puma robot arm were simulated. Each of 
these degrees of freedom were controlled simultaneously using two joysticks. 
Although no experiments have yet been performed with the puma simulation, it 
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Ames-Berkeley Visual Enhancement Display 

Figure 2 
- 6 - 



is hoped that it will be a step toward experiments with more complex 
manipulators. A low-bandwidth telephone connection to control two puma arms 
at Jet Propulsion Labs in Pasadena is planned. The simulation will allow 
prediction of the robots motion to provide a preview display to help overcome 
the communication delays inherent in such a low bandwidth connection, or as in 
transmissions to manipulators in space. 

CONTROL MODE EXPERIMENTS 


Position and rate controls are the two common manual control modes 
for controlling telemanipulators with joysticks (or hand controllers) (Johnsen 
& Corliss, 1971; Heer, 1973). In the position control the joystick command 
indicates the desired end effector position of the manipulator, whereas in the 
rate control the joystick command indicates the desired and effector velocity. 

In our three-axis pick-and-pl ace tasks, the human operator controls 
the manipulator hand position in the robot base Cartesian coordinate by using 
three axes of the two displacement joysticks. In pure (or ideal) position 
control, the system transfer function from the joystick displacement input to 
the actual manipulator hand position output is a constant gain Gp for each 
axis. In pure rate control, the system transfer function is a single 
integrator G^/s for each axis. In the rate control, a 5 % dead-band 
nonlinearity is introduced before the pure integrator in order to inhibit the 
drift problem associated with the pure integrator. 
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Comparison of pure position and rate controls 

Three-axis pick-and-place tasks were performed with both pure 
position and rate control modes for various gains (Figure 3). The mean 
completion time plot clearly shows that pick-and-place performance with pure 
position control (mean completion time 2.8 seconds at G p = 2) was about 1.5 
times faster than that of the pure rate control (mean completion time 4.3 
seconds at G y = 4). 


Traj ectori es of Joystick and Mani pul ator Movements 

In order to examine why the position control performed better than 
the rate control, several trajectories of the joystick displacement input and 
the manipulator hand position output during the pick-and-place operation were 
observed. Typical trajectories from the start of trying to pick up an object 
to its accomplishment were plotted to illustrate position, rate, and 
acceleration controls (Figure 4). Components only for the x-axis 
(side-to-side) are plotted, since components for the other two axes are 
similar. Observation of several trajectories indicates that a precise 
re-positioning of the manipulator hand is achieved by a combination of quick 
step re-positioning operations and slow smooth movement operations. In 
position control one quick step re-positioning of the manipulator hand from 
one position to another requires one joystick pull or push operation, whereas 
in the rate control it requires a pair of operations; pull-and-push or 
push-and-pul 1 operations (Figure 4). This is a major reason why the position 
control yielded better performance than the rate control for our 
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pick-and-place tasks. It should be noted, however, that the pick-and-place 
task is a positioning task. If the task is following a target with a constant 
velocity, then velocity (rate) control would perform better. 


Acceleration Control 

Three-axis pick-and-place task were also tried with acceleration 
control. It turned out, however, acceleration control was not adequate to 
perform stable, safe pick-and-place operations. In acceleration control, the 
manipulator tends to move almost all the time even though the joystick is at 
the center position. Note that in pure rate control, the manipulator does not 
move when the joystick is at the center position regardless of previous 
history of the joystick displacement. 

Human Adaptation to Gain Change 

Mean completion time did not change much for the various gains tested 
(Figure 3), which means that the human operator adapted well to the gain 
change (McRuer, et al , 1965; Young, 1969; Stark 1968). Both lower and higher 
gains relative to the optimal gains caused slight increase in the mean 
completion time. A reason of slightly longer mean completion times with lower 
gains is because lower gains demand wider joystick displacements and it takes 
longer for the finger or hand to displace the joystick wider. A reason for 
slightly longer mean completion times with higher gains is the higher gains 
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demand more minute joystick displacements, degrading effective resolution of 
the joystick control. An additional major reason for longer mean completion 
times with lower gains for the rate control is due to the velocity limit. 

Force joystick 

The two common joystick types are the displacement and force 
joysticks. The output of the displacement joystick is proportional to the 
joystick displacement, whereas the output of the force joystick (isometric or 
stiff joystick) is proportional to the force applied by the human operator. 
The advantage of the force joystick is that it requires only minute joystick 
displacements (a few micrometers) in contrast with the displacement joystick 
(a few centimeters). 

Pick-and-pl ace tasks wee* performed for pure position and rate 
controls with displacement and force joysticks. The experimental results for 
two subjects (Figure 5) shows that in the rate control, task performance with 
force joystick was significantly faster than that with displacement joystick. 
This is mainly because the force joystick senses the applied force directly, 
requiring only very minute joystick displacements. In the position control, 
however, the force joystick performed no better than the displacement 
joystick. In fact, all three subjects preferred to use the displacement 
joystick in this mode, since the force joystick required more force to be 
applied than the displacement joystick , especially when the manipulator hand 
is to be positioned far away from the initial center position. Position 
control also performed better than the rate control regardless of joystick 
types, and furthermore the position control with the displacement joystick 
performed best for our pick-and-pl ace tasks (Figure 5). 
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Resolution 


The experimental results demonstrate the superiority of position 
control when the telemanipulator has a sufficiently small work space (Figures 
3, 4, 45). Note that our three-axis pick-and-place tasks used in this 
experiment implicitly assumes that the manipulator work space is small or at 
least not very large, since our task allows the human operator to perform 
successful pick-and-place operations with a display showing the entire work 
space on the screen. Examples of small work space telemanipulators can be 
found in nuclear reactor teleoperators, surgical micro-telerobots, or small 
dexterous telerobotic hands. Position control can also be utilized during 
proximity operations in conjunction with the force-reflecting joysticks for 
enhanced telepresence (Bejczy, 1980). When the telemanipulator's work space 
is very large as compared to human operator's control space, position control 
of the entire work space suffers from poor resolution since human operator's 
control space must be greatly up-scaled to acconnodate the telemanipulator's 
large work space (Flatau, 1973). One way of solving this poor resolution 
problem in position control is using indexing Johnsen 4 Corliss, 1971; Argonne 
National Lab, 1967). In the indexed position control mode, the control stick 
gain is selected so that the full displacement range of the control stick can 
cover only a small portion of the manipulator work space, and large movements 
of the manipulator hand can be made by successive uses of an indexing trigger 
mounted on the control stick. Note, however, that rate control can inherently 
provide any higher degree of resolution by mere change of control stick gain 
without use of indexing. 
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HOMEOMORPHIC CONTROLLER 


Most of our pick-and-pl ace and tracking experiments were performed 
with joysticks as the input device through which the human operator controlled 
the simulated manipulator. The operator's movements when using joysticks are 
non-homeomorphic , so that the movements he must make to produce a desired 
manipulator response do not match the movement of the manipulator and 
effector. Thus, he must mentally convert the desired end effector position to 
Cartesian coordinates and use the joysticks to input these coordinates. 

To attempt to study whether a truly homeomorphic input device could 
improve performance in tracking tasks, an apparatus of identical form to our 
simulated cylindrical manipulator was built. A vertical rod was supported by 
bearings on the base to allow rotation, theta. A counterweighted horizontal 
arm was attached to the rod with sliding bearings to permit rotation and 
translation in the r and z axes respectively. The human operator could 
control position through a handle on the end of the arm corresponding to the 
end effector of the simulated manipulator. Potentiometers measured movement 
in each axis to determine input r, theta, and z. The LSI -11/23 computer read 
these values through A/D channels and displayed the manipulator in the 
identical position. 

Three-dimensional tracking experiments were performed with the 
homeomorphic controller and with joysticks for gains varying from 1 to 5 to 
compare performance (Figure 6). The results do not show a significant 
difference between the homeomorphic controller and joysticks over the range of 
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gain values. Although the larger movements required for the homeomorphic 
controller, with greater inertia and friction than the joystick, may have 
limited performance, we believe that human adaptability minimizes its 
advantages. 


TRAINING BY OPTIMAL CONTROL EXAMPLE 

A simplified simulation of the manned maneuvering unit, MMV, enabled 
study of training of human control performance (Jordan, 1985). Only three 
translatory degrees-of-freedom, x, y and z, were used. Thrusters generating 
pulses of acceleratory control were controlled via a keyboard and the task was 
to accelerate simultaneously in x, y and z to a maximum velocity, transit to 
the desired new location, and decelerate again simultaneously. Two displays 
were used — a perspective display of a minified model of the MMV, or two 
two-dimensional projectors of that model with a small inset of the perspective 
display. 

Subjects generally performed poorly during the few hundred seconds 
allowed for the tasks (upper panels. Figure 7). It was decided to allow the 
subjects to view this control problem carried out by a simple optimal control 
algorithm (see middle panel, Figure 7). This experience was of considerable 
help and several subjects then performed quite well (bottom panel, Figure 7). 

This experiment, learning-by-example, illustrates a strategy that 
perhaps may be effective in more complex and realistic tasks as well. 
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COMMUNICATIONS DELAY AND PREVIEW 


Communicati on delay is a significant constraint in human performance 
in controlling a remote manipulator. It has been shown (Sheridan et al , 1964, 
1966; Tomizuka and Whitney, 1976) that preview information can be used to 
improve performance. Stark et al (1986) demonstrated that preview can 
significantly reduce error in tracking experiments with imposed delay. 

Experiments were performed to investigate whether a preview display 
could improve performance in pick-and-place tasks with delay. A single bright 
diamond- shaped cursor was added to the display to represent current joystick 
position. This was a perfect prediction of what the end effector position 
would be after the delay interval. Thus, the task was the same as if there 
were no delay, except that the HO had to wait one delay period for 
confirmation that a target had been touched or correctly placed (in the 
non-previewed display, the target letter was doubled when picked up, and 
became single again when placed in the correct box). 

Performance affected by 

Preview improved performance at delays up to 4 seconds so that it was 
almost as good as for a small delay of 0 2 seconds (Figure 8). While task 
completion time in the delayed condition increased greatly with delay, there 
was only a small increase in the preview case. This is because the H.O. must 
compensate for delays by using a "move-and-wait" strategy, making a joystick 
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movement and waiting to see the resultant end effector movement. In the 
preview case, this strategy is only necessary when very close to the target or 
box to wait for confirmation that the goal has indeed been touched. 


HELMET MOUNTED DISPLAYED DESIGN 


Motivation 

The motivation of the HMD system is to provide the human operator 
with a telepresence feeling that he is actually in the remote site and 
controls the telemanipulator directly. The HMD system detects the human 
operator's head motion, and controls the remote stereo camera accordingly. In 
our current system, the remote telemanipulation task environment is simulated 
and the pictures for the display are generated by the computer. 

Head Orientation Sensors 

A two-axis magnetic Helmholtz coil arrangement was used as a head 
orientation sensing device, to detect horizontal and vertical head rotations 
(Figure 9). By assuming that the pan and tilt angles of a remote stereo 
camera are controlled in accordance with the horizontal and vertical head 
rotations, respectively, the computer generated the corresponding stereo 
picture for the HMD. The head orientation sensing device is composed of a 
search (sensing) coil mounted on or beneath the helmet and two pairs of field 
coils fixed with respect to human operator's control station. The right-left 
pair of the field coil generates the horizontal magnetic flux of a 50 KHz 
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square wave. The up-down pair of the field coil generates the vertical 
magnetic flux of a 75 KHz square wave. The search coil detects the induced 
magnetic flux, which is amplified and separated into 50 and 75 KHz 
components. The magnitude of each frequency component depends upon the 
orientation of the search coil with respect to the corresponding field coil 
(Duffy, 1985). 


LCD Display 

An early configuration of the HMD had a flat-panel LCD (liquid 
crystal display) screen (a cotimercially available portable LCD television) 
mounted on the helmet for the display (Figure 10). However, the picture 
quality of the LCD screen was poor due not only to low resolution but also to 
poor contrast. 


CRT Display 

A new design of the HMD that we currently have, mounts a pair of Sony 
viewfinders (Model VF-208) on the helmet (Figure 5). Each viewfinder has a 
1-inch CRT (cathode ray tube) screen and converging lens through which the 
human operator views the CRT screen. The computer-generated stereo picture 
pair (stereogram) is displayed on the CRT screens; one for the left eye and 
the other for the right. The converging lens forms the virtual image of the 
stereogram behind the actual display screen. When the CRT screen is 4.2 cm 
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apart frois whose focal length is 5 cm, the virtual image of the CRT 
screen 1st 25 cm apart from the lens with an image magnification of 
6. Thus, CRT screen appears to be a 6-inch screen to the viewer. At 
appropriatrical and optical conditions, the right and left images 
overlay, aaeople can fuse the two images into a single 
three-dimeimage. The stereoscopic display formulas used to generate 
the stereo the helmet mounted display are described in references 
(Kim, et a;ted 1985). 

Mechanical 

Res of freedom were provided for the mechanical adjustment 
of the posi orientation of each viewfinder, allowing three orthogonal 
slidings artations (Figure 11). A 1 lb. counterweight was attached to 
the back ofmet for counterbalancing. 


SUMMARY 


Thsxperiments enabled our Telerobotic Unit at the University 
of CalifornVley, to explore in a number of research directions. The 
HMD directiQw been greatly extended and is a major focus in our 
laboratory, other hand, the homeomorphic controller did not seem to be 
a productive; to continue because of the adaptability of the H.O. to 
many configuof control. Also, our interest in supervisory and other 
high level cjs leading us away from direct manual control. The 
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ithusiastic and felt the course stimulated their creativity and 
irtunity for them to engage in relatively unstructured 
— a good model for subsequent thesis research. 
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